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1. INTRODUCTION 


The Progress in International Reading Literacy Study (PIRLS) is a large international 
comparative study of the reading literacy of young students. The student population for the U.S. 2001 
PIRLS (hereafter simply referred to as PIRLS) was the set of all fourth-graders in the United States, 
corresponding to the grade in which the highest proportion of nine -year-olds are enrolled. The PIRLS 
school sample consisted of 200 schools (150 public and 50 private) containing a fourth grade, selected 
with probability proportionate to the school’s enrollment of fourth-graders. One classroom was sampled 
from each selected school. 

PIRLS was conducted in April and May 2001. For the original sample, the unweighted 
response rate at the school level was 62.5 percent, with 125 out of 200 schools responding. Through the 
use of replacements, the unweighted response rate was improved to 87 percent, with 174 out of 200 
schools responding. However, as the response rate from the original sample was below 85 percent, NCES 
requested that Westat investigate the potential magnitude of nonresponse bias at the school level. The 
methodology and results of this investigation follow. 


2. METHODOLOGY 


There are at least two possible ways to analyze nonresponse bias given that replacement 
schools were used as substitutes for schools from the original sample that did not respond. One method is 
to base the analysis exclusively on the original sample of 200 schools and to treat all those that were 
substituted as nonrespondents. A second method is to base the analysis on the final sample of 200 schools 
(including replacements) and to treat as nonrespondents those schools from whom a final response was 
not received. The results of the first method are presented in section 3.1 of this report, while the results of 
the second method are contained in section 3.2. 

In order to compare PIRLS respondents and nonrespondents it was necessary to match the 
sample of schools back to the sample frame to pick up as many characteristics as possible that might 
provide information about the presence of nonresponse bias. Comparing frame characteristics for 
respondents and nonrespondents is not always a good measure of nonresponse bias if the characteristics 
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are unrelated or weakly related to more substantive items in the survey, however this is often the only 
approach available. Frame characteristics were taken from the 1997-98 Common Core of Data (CCD) for 
public schools, and from the 1997-98 Private School Survey (PSS) for private schools. For categorical 
variables, response rates by characteristic were calculated. The hypothesis of independence between the 
characteristic and response status was tested using a Rao-Scott modified Chi-square statistic. For 
continuous variables, summary means were calculated. The 95 percent confidence interval for the 
difference between the mean for respondents and the mean for nonrespondents was tested to see whether 
or not it included zero. In addition to these tests, logistic regression models were set up to identify 
whether any of the frame characteristics were significant in predicting response status. All analyses were 
performed using WesVar and replicate weights to properly account for the complex sample design. The 
base weights used did not include a nonresponse adjustment factor. Due to the lack of primary sampling 
unit (PSU) information on the files received from the school sampling contractor, it was necessary to 
create replicate weights in WesVar assuming a two-stage design (schools, and classrooms within 
schools). The JK2 method was used, and the RS3 statistic was used for the Chi-square tests. 


3. RESULTS 


3.1 Original Sample 

The following nonresponse bias analysis is based exclusively on the original sample of 200 
schools. All schools that were substituted by a replacement were treated as nonrespondents, as were any 
nonresponding original schools that were not substituted. Standard errors are given throughout in 
parentheses. 


Of initial interest was the relationship between response status and whether the school was 
public or private. Table 1 shows the relevant response rates. The test of independence gives RS3 = 0.403, 
with a p-value of 0.526. This indicates that there is no significant relationship between response status and 
public/private at the 5 percent level. 
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Table 1. Original sample school response rate, by public/private and overall 


Category 


Response rate 

Estimate (%) 

Standard error (%) 

Total 

61.20 

(6.302) 

Public 

64.31 

(4.973) 

Private 

53.49 

(14.698) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


3.1.1 Categorical V ariables 

The following characteristics were available for both public and private schools. 

■ Community type 

■ Public/religious affiliation 

■ Census region 

Table 2 shows school response rates by community type. The test of independence gives 
RS3 = 0.523, with a p-value of 0.649. This indicates that there is no significant relationship between 
response status and community type at the 5 percent level. 


Table 2. Original sample school response rate, by community type 


( 


Response rate 

U/dLegory — 

Estimate (%) 

Standard error (%) 

Central city 

68.84 

(6.518) 

Urban fringe or large town 

56.86 

(7.619) 

Rural or small town 

61.00 

(11.393) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


Table 3 shows school response rates by public/religious affiliation. The test of independence 
gives RS3 = 4.823, with a p-value of 0.072, however this must be interpreted with caution due to the 
presence of a cell with less than five observations. There is some evidence that Catholic schools were 
more likely to respond than others, but it is not significant at the 5 percent level. 
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Table 3. Original sample school response rate, by public/religious affiliation 

Category 


Response rate 

Estimate (%) 

Standard error (%) 

Public 

64.31 

(4.973) 

Private — Catholic 

90.09 

(6.974) 

Private — Other religious 

20.54 

(14.063) 

Private — Non-sectarian 

78.88 

(39.113) 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 

Table 4 shows 

school response rates by census 

region. The test of independence gives 

RS3 = 1.063, with a p-value of 0.624. This indicates that there is no significant relationship between 

response status and census region at the 5 percent level. 


Table 4. Original sample school response rate, by census region 

Category 


Response rate 

Estimate (%) 

Standard error (%) 

Northeast 

58.98 

(9.708) 

Midwest 

73.67 

(8.308) 

South 

58.04 

(11.549) 

West 

59.60 

(7.549) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


3.1.2 Continuous Variables 

The following characteristics were available for both public and private schools. 

■ Number of students enrolled in grade 4 

■ Total number of students 

■ Percentage Asian or Pacific Islander students 

■ Percentage Black, non-Hispanic students 

■ Percentage Hispanic students 

■ Percentage American Indian or Alaska Native students 
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Percentage White, non-Hispanic students 

Ratio of total students to full-time equivalent (FTE) teachers 


Table 5 shows the mean number of grade 4 students and the mean total number of students 
for responding and nonresponding schools. 


Table 5. Mean grade 4 enrollment and total students for original sample schools, by response status 


( olormri/ . 

Responding 


Nonresponding 

U/dLegory 

Estimate Standard error 

Estimate 

Standard error 

Total number of students 

415.17 


386.32 

(65.155) 

Students enrolled in grade 4 

60.78 

(4.754) 

58.93 

(10.794) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


The difference in the mean grade 4 enrollment is 1.85, with a 95 percent confidence interval 
of (-22.23, 25.92). The confidence interval includes zero, therefore there is no evidence that the mean 
grade 4 enrollment of responding and nonresponding schools is significantly different at the 5 percent 
level. 


The difference in the mean total students is 28.86, with a 95 percent confidence interval of 
(-1 15.64, 173.35). The confidence interval includes zero, therefore there is no evidence that the mean total 
enrollment of responding and nonresponding schools is significantly different at the 5 percent level. 


schools. 


Table 6 shows the mean race/ethnicity percentages for responding and nonresponding 


The difference in the mean percentage of Asian or Pacific Islander students is -0.35 percent, 
with a 95 percent confidence interval of (-2.31 percent, 1.60 percent). The confidence interval includes 
zero, therefore there is no evidence of a significant difference in the mean percentage of Asian or Pacific 
Islander students at the 5 percent level. 

The difference in the mean percentage of Black, non-Hispanic students is 0.82 percent, with 
a 95 percent confidence interval of (-8.98 percent, 10.61 percent). The confidence interval includes zero, 
therefore there is no evidence of a significant difference in the mean percentage of Black, non-Hispanic 
students at the 5 percent level. 
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Table 6. Mean race/ethnicity percentages for original sample schools, by response status 


Category 

Responding 

Nonresponding 

Estimate (%) 

Standard 
error (%) 

Estimate (%) 

Standard 
error (%) 

Asian or Pacific Islander students 

2.68 

(0.640) 

3.03 

(0.759) 

Black, Non-Hispanic students 

13.60 

(3.342) 

12.79 

(3.497) 

Hispanic students 

9.72 

(1.915) 

8.87 

(2.063) 

American Indian or Alaska Native students 

2.89 

(2.211) 

0.52 

(0.175) 

White, Non-Hispanic students 

71.06 

(4.299) 

74.74 

(5.290) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


The difference in the mean percentage of Hispanic students is 0.85 percent, with a 95 percent 
confidence interval of (-4.80 percent, 6.50 percent). The confidence interval includes zero, therefore there 
is no evidence of a significant difference in the mean percentage of Hispanic students at the 5 percent 
level. 


The mean percentage of American Indian or Alaska Native students is 2.37 percent, with a 
95 percent confidence interval of (-2.02 percent, 6.75 percent). The confidence interval includes zero, 
therefore there is no evidence of a significant difference in the mean percentage of American Indian or 
Alaska Native students at the 5 percent level. 

The mean percentage of White, non-Hispanic students is -3.68 percent, with a 95 percent 
confidence interval of (-17.38 percent, 10.01 percent). The confidence interval includes zero, therefore 
there is no evidence of a significant difference in the mean percentage of White, non-Hispanic students at 
the 5 percent level. 

Table 7 shows the mean ratio of total students to FTE teachers for responding and 
nonresponding schools. The difference in means is 2.94, with a 95 percent confidence interval of (-0.19, 
6.06). The confidence interval includes zero, therefore there is no evidence of a significant difference in 
the mean ratio of total students to FTE teachers for responding and nonresponding schools, at the 5 
percent level. 
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Table 7. 


Mean ratio of total students to FTE teachers for original sample schools, by response status 


Responding 

Nonresponding 

Category 

„ . ^ Standard 

Estimate 

error 

Estimate 

Standard 

error 

Ratio of total students to FTE teachers 

16.15 (0.750) 

13.21 

(1.346) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


For public schools only, another characteristic was available. 


■ Percentage of students eligible to participate in Free Lunch Program under the 
National School Lunch Act 

Table 8 shows the mean percentage of students eligible for the Free Lunch Program for 
responding and nonresponding public schools. The difference in means is -6.66 percent, with a 95 percent 
confidence interval of (-18.53 percent, 5.21 percent). The confidence interval includes zero, however this 
must be interpreted with caution because the “free lunch” variable itself is missing for 35 out of the 150 
public schools. The result suggests that the mean percentage of students eligible for the Free Lunch 
Program is not significantly different for responding and nonresponding public schools, at the 5 percent 
level. 

Table 8. Mean percentage of students eligible for Free Lunch Program for original sample schools, by 
response status: Public schools only 



Responding 

Nonresponding 

Category 

r, .. . , 0/ . Standard 

Estimate (%) /n/ . 

error (%) 

„ , 0/ . Standard 

Estimate (%) 

error (%) 

Students eligible for Free Lunch 
Program 

34.10 (4.053) 

40.76 (4.673) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


For private schools only, the following characteristics were available. 

■ Number of FTE teachers 

■ Percent male students 

Table 9 shows the mean number of FTE teachers responding and nonresponding private 
schools. The difference in means is -3.27, with a 95 percent confidence interval of (-14.31, 7.78). The 
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confidence interval includes zero, therefore there is no evidence of a significant difference in the mean 
number of FTE teachers at the 5 percent level. 


Table 9. Mean number of FTE teachers for original sample schools, by response status: Private 
schools only 



Responding 


Nonresponding 

Category 

Estimate 

Standard 

error 

Estimate 

Standard 

error 

FTE teachers 

13.76 

(2.116) 

17.02 

(5.141) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


Table 10 shows the mean percentage of male students for responding and nonresponding 
private schools. The difference in means is -8.06 percent, with a 95 percent confidence interval of (-13.71 
percent, -2.41 percent). The confidence interval does not include zero, therefore there is evidence that the 
mean percentage of male students is lower for responding private schools at the 5 percent level of 
significance. 


Table 10. Mean percentage of male students for original sample schools, by response status: Private 
schools only 



Responding 


Nonresponding 

Category 

Estimate (%) 

Standard 
error (%) 

t~. , • . /n/ . Standard 

Estimate (%) /n/ . 

error (%) 

Male students 

50.42 

(1.614) 

58.48 (2.277) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


This result indicates a potential source of bias in the PIRLS survey results for private 
schools, related to gender composition of school. Unfortunately this characteristic was not available for 
analysis for public schools. 


3.1.3 Logistic Regression Model 


A logistic regression model was set up treating response status as the binary dependent 
variable and frame characteristics as the predictor variables. Response was treated as “success” and 
nonresponse as “failure.” 















Public and private schools were modeled together using the following 1 1 variables. 

■ Community type 

■ Public/religious affiliation 

■ Census region 

■ Number of students enrolled in grade 4 

■ Total number of students 

■ Percentage Asian or Pacific Islander students 

■ Percentage Black, non-Hispanic students 

■ Percentage Hispanic students 

■ Percentage American Indian or Alaska Native students 

■ Percentage White, non-Hispanic students 

■ Ratio of total students to FTE teachers 


Initial model fitting was performed in SAS in order to make use of the stepwise model 
selection option. The only predictor variable to make it into the final model was public/religious 
affiliation. This model was refitted using WesVar to take proper account of the complex sample design 
and confirmed to be the most parsimonious model. The final estimated model was as follows. 


c 

log 

V 


.P(Response) 
/’(Non - response y 


= 1.318 - 0.729 * Public + 0.890 * Catholic - 2.671 * Other Religious 


In the above equation, “Public,” “Catholic,” and “Other Religious” are mutually exclusive 
indicator variables of the implied school characteristics. The negative “Public” and “Other Religious” 
parameter estimates indicate that public and other religious schools were less likely to respond to PIRLS. 
The positive “Catholic” parameter estimate indicates that Catholic schools were more likely to respond to 
PIRLS. Standard errors and tests of hypotheses for the model parameter estimates are presented in 
table 1 1 . 
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Table 11. Final model parameters for original sample schools 


Parameter 

Estimate 

Standard error 

Test for HO: Parameter = 0 

P-value 

Intercept 

1.318 

1.7674 

0.7457 

0.4576 

Public 

-0.729 

1.7806 

-0.4095 

0.6831 

Catholic 

0.890 

1.9936 

0.4463 

0.6564 

Other religious 

-2.671 

2.0857 

-1.2805 

0.2033 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


When the model is fit in WesVar using correct standard error estimates, the p-values above 
indicate that there is no significant difference between the effect of the (omitted) reference category, 
private-non-sectarian, and any of the other three categories. However, the F-value measuring the overall 
fit of the model is 5.1684, with a p-value of 0.0023. This indicates that the public/religious affiliation 
characteristic is a significant predictor of the response status of schools at the 5 percent level of 
significance. This apparent contradiction is easily explained away by looking at an alternative 
parameterization of the model, where Catholic is treated as the reference category. Such an analysis 
shows that there is a significant difference in effect when Catholic is compared to public, or to private- 
other religious. 


3.2 Final Sample 

The following nonresponse bias analysis is based on the final sample of 200 schools, 
including replacements. All schools from whom a final response was not received were treated as 
nonrespondents. Through the use of replacements, the unweighted response rate was improved to 87 
percent, with 174 out of 200 schools responding. Standard errors are given throughout in parentheses. 

Of initial interest was the relationship between response status and whether the school was 
public or private. Table 12 shows the relevant response rates. The test of independence gives 
RS3 = 1.865, with a p-value of 0.172. This indicates that there is no significant relationship between 
response status and public/private at the 5 percent level. 
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Table 12. Final sample school response rate, by public/private and overall 


Category 


Response rate 

Estimate (%) 

Standard error (%) 

Total 

91.97 

(1.883) 

Public 

90.42 

(2.313) 

Private 

95.64 

(2.677) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


3.2.1 Categorical Variables 

The following characteristics were available for both public and private schools. 

■ Community type 

■ Public/religious affiliation 

■ Census region 

Table 13 shows school response rates by community type. The test of independence gives 
RS3 = 3.369, with a p-value of 0.180. This indicates that there is no significant relationship between 
response status and community type at the 5 percent level. 


Table 13. Final sample school response rate, by community type 




Response rate 

UxdLcgory — 

Estimate (%) 

Standard error (%) 

Central city 

87.85 

(4.416) 

Urban fringe or large town 

88.35 

(4.043) 

Rural or small town 

95.40 

(2.238) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


Table 14 shows school response rates by public/religious affiliation. The RS3 test statistic 
cannot be computed because the table contains a cell with zero observations. The ordinary Pearson Chi- 
square test statistic (that does not take into account the complex sample design) equals 1.716, with a 
p-value of 0.633. This must also be interpreted with caution due to the presence of a cell with less than 
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five observations, however it would suggest that there is no significant relationship between response 
status and public/religious affiliation at the 5 percent level. 


Table 14. Final sample school response rate, by public/religious affiliation 


Category 


Response rate 

Estimate (%) 

Standard error (%) 

Public 

90.42 

(2.313) 

Private — Catholic 

95.72 

(4.096) 

Private — Other religious 

94.81 

(3.581) 

Private — N on-sectarian 

100.0 

(0.0) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


Table 15 shows school response rates by census region. The test of independence gives 
RS3 = 2.348, with a p-value of 0.485. This must be interpreted with caution due to the presence of a cell 
with less than five observations, however it would suggest that there is no significant relationship 
between response status and census region at the 5 percent level. 


Table 15. Final sample school response rate, by census region 


Category 


Response rate 

Estimate (%) 

Standard error (%) 

Northeast 

91.39 

(4.079) 

Midwest 

93.61 

(4.401) 

South 

94.14 

(2.316) 

West 

86.18 

(5.176) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


3.2.2 Continuous Variables 

The following characteristics were available for both public and private schools. 

■ Number of students enrolled in grade 4 

■ Total number of students 

■ Percentage Asian or Pacific Islander students 
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■ Percentage Black, non-Hispanic students 

■ Percentage Hispanic students 

■ Percentage American Indian or Alaska Native students 

■ Percentage White, non-Hispanic students 

■ Ratio of total students to FTE teachers 

Table 16 shows the mean number of grade 4 students and the mean total number of students 
for responding and nonresponding schools. 


Table 16. Mean grade 4 enrollment and total students for final sample schools, by response status 


Responding 

Nonresponding 

Category 

Estimate 

Standard 

error 

Estimate 

Standard 

error 

Total number of students 

385.27 

(31.822) 

605.36 

(40.449) 

Students enrolled in grade 4 

55.19 

(5.162) 

98.02 

(7.916) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


The difference in the mean grade 4 enrollment is -42.83, with a 95 percent confidence 
interval of (-62.38, -23.28). The confidence interval does not include zero, therefore there is evidence that 
the mean grade 4 enrollment is lower for responding schools at the 5 percent level of significance. 

The difference in the mean total students is -220.09, with a 95 percent confidence interval of 
(-328.05, -112.13). This confidence interval also excludes zero, therefore there is evidence that the mean 
total enrollment is lower for responding schools at the 5 percent level of significance. 

These results indicate a potential source of bias in the P1RLS survey results, related to size of 

school. 


schools. 


Table 17 shows the mean race/ethnicity percentages for responding and nonresponding 
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Table 17. Mean race/ethnicity percentages for final sample schools, by response status 



Responding 

Nonresponding 

Category 

Estimate (%) 

Standard 
error (%) 

Estimate (%) 

Standard 
error (%) 

Asian or Pacific Islander students 

2.86 

(0.501) 

4.32 

(1.492) 

Black, Non-Hispanic students 

14.22 

(2.336) 

13.57 

(4.147) 

Hispanic students 

10.27 

(1.779) 

12.90 

(4.057) 

American Indian or Alaska Native students 

1.94 

(1.468) 

1.26 

(0.775) 

White, Non-Hispanic students 

70.67 

(3.128) 

67.95 

(6.439) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


The difference in the mean percentage of Asian or Pacific Islander students is -1.46 percent, 
with a 95 percent confidence interval of (-4.62 percent, 1.71 percent). The confidence interval includes 
zero, therefore there is no evidence of a significant difference in the mean percentage of Asian or Pacific 
Islander students at the 5 percent level. 


The difference in the mean percentage of Black, non-Hispanic students is 0.65 percent, with 
a 95 percent confidence interval of (-9.19 percent, 10.50 percent). The confidence interval includes zero, 
therefore there is no evidence of a significant difference in the mean percentage of Black, non-Hispanic 
students at the 5 percent level. 


The difference in the mean percentage of Hispanic students is -2.63 percent, with a 95 
percent confidence interval of (-11.58 percent, 6.32 percent). The confidence interval includes zero, 
therefore there is no evidence of a significant difference in the mean percentage of Hispanic students at 
the 5 percent level. 


The difference in the mean percentage of American Indian or Alaska Native students is 0.68 
percent, with a 95 percent confidence interval of (-2.41 percent, 3.78 percent). The confidence interval 
includes zero, therefore there is no evidence of a significant difference in the mean percentage of 
American Indian or Alaska Native students at the 5 percent level. 


The difference in the mean percentage of White, non-Hispanic students is 2.72 percent, with 
a 95 percent confidence interval of (-11.07 percent, 16.51 percent). The confidence interval includes zero, 
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therefore there is no evidence of a significant difference in the mean percentage of White, non-Hispanic 
students at the 5 percent level. 


Table 18 shows the mean ratio of total students to FTE teachers for responding and 
nonresponding schools. The difference in means is -2.39, with a 95 percent confidence interval of (-5.47, 
0.68). The confidence interval includes zero, therefore there is no evidence of a significant difference in 
the mean ratio of total students to FTE teachers for responding and nonresponding schools, at the 5 
percent level. 

Table 18. Mean ratio of total students to FTE teachers for final sample schools, by response status 


| CltAfTArU 

Responding 

Nonresponding 

V^/d-LCgUiy 

Estimate 

Standard error 

Estimate 

Standard error 

Ratio of total students to FTE 
teachers 

15.69 

(0.674) 

18.08 

(1.231) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


For public schools only, another characteristic was available. 

■ Percentage of students eligible to participate in Free Lunch Program under the 
National School Lunch Act 

Table 19 shows the mean percentage of students eligible for the Free Lunch Program for 
responding and nonresponding public schools. The difference in means is -9.66 percent, with a 95 percent 
confidence interval of (-19.66 percent, 0.34 percent). The confidence interval only just includes zero, 
however this must be interpreted with caution because the “free lunch” variable itself is missing for 35 
out of the 150 public schools. The result suggests that the mean percentage of students eligible for the 
Free Lunch Program is not significantly different for responding and nonresponding public schools, at the 
5 percent level. 
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Table 19. Mean percentage of students eligible for Free Lunch Program for final sample schools, by 
response status: Public schools only 



Responding 

Nonresponding 

Category 

r- .. . /0/ , Standard 

Estimate (%) 

error (%) 

r .. , Standard 

Estimate (%) /n/ . 

error (%) 

Students eligible for Free Lunch 
Program 

37.97 (3.136) 

47.63 (3.741) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


For private schools only, the following characteristics were available. 

■ Number of FTE teachers 

■ Percentage of male students 

Table 20 shows the mean number of FTE teachers responding and nonresponding private 
schools. The difference in means is -22.18, with a 95 percent confidence interval of (-45.44, 1.08). The 
confidence interval only just includes zero. There is some evidence that the mean number of FTE teachers 
is lower for responding private schools, though it is not significant at the 5 percent level. 


Table 20. Mean number of FTE teachers for final sample schools, by response status: Private schools 
only 


Category 

Responding 

Nonresponding 

Estimate 

Standard error 

Estimate 

Standard error 

FTE teachers 

11.96 

(2.018) 

34.14 

(11.547) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


Table 21 shows the mean percentage of male students for responding and nonresponding 
private schools. The difference in means is 3.23 percent, with a 95 percent confidence interval of (0.16 
percent, 6.31 percent). The confidence interval does not include zero, therefore there is evidence that the 
mean percentage of male students is lower for responding private schools at the 5 percent level of 
significance. 
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Table 21. Mean percentage of male students for final sample schools, by response status: Private schools 
only 



Responding 


Nonresponding 

Category 

Estimate (%) 

Standard 
error (%) 

„ .. . /0/ , Standard 

Estimate (%) 

error (%) 

Male students 

50.42 

(1.095) 

47.18 (1.206) 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


This result indicates a potential source of bias in the P1RLS survey results for private 
schools, related to gender composition of school. Unfortunately this characteristic was not available for 
analysis for public schools. 


3.2.3 Logistic Regression Model 

A logistic regression model was set up treating response status as the binary dependent 
variable and frame characteristics as the predictor variables. Response was treated as “success” and 
nonresponse as “failure.” 

Public and private schools were modeled together using the following 1 1 variables. 

■ Community type 

■ Public/religious affiliation 

■ Census region 

■ Number of students enrolled in grade 4 

■ Total number of students 

■ Percentage Asian or Pacific Islander students 

■ Percentage Black, non-Hispanic students 

■ Percentage Hispanic students 

■ Percentage American Indian or Alaska Native students 

■ Percentage White, non-Hispanic students 
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Ratio of total students to FTE teachers 


Initial model fitting was performed in SAS in order to make use of the stepwise model 
selection option. The only predictor variable to make it into the final model was grade 4 enrollment. This 
model was refitted using WesVar to take proper account of the complex sample design and confirmed to 
be the most parsimonious model. The final estimated model was as follows. 


/ 

log 

V 


.P(Response) 

P( Nonresponse y 


= 3.822 - 0.019 * Number of students enrolled in grade 4 


The negative “Number of students enrolled in grade 4” estimate indicates that schools with a 
higher number of students in grade 4 were less likely to respond to PIRLS. Standard errors and tests of 
hypotheses for the model parameter estimates are presented in table 22. 


Table 22. Final model parameters for final sample schools 


Parameter 

Estimate 

Standard error 

Test for HO: 
Parameter = 0 

P-value 

Intercept 

3.822 

0.4420 

8.6471 

<0.0001 

Numher of students 

enrolled in grade 4 

-0.019 

0.0037 

-5.0338 

<0.0001 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study, 2001. 


The F-value measuring the overall fit of the model is 25.34, with a p-value < 0.0001. This 
indicates that the number of students enrolled in grade 4 is a significant predictor of the response status of 
schools, even at the 1 percent level of significance. This finding is consistent with the statistically 
significant difference in mean grade 4 enrollment by response status, considered previously. 


3.2.4 Size of School and Reading Literacy 

Given the findings presented earlier, it is important to question whether the substantive 
results of the survey differ according to size of school. (Obviously this relationship can only be analyzed 
for respondents.) If they do not, then there is less cause for concern over nonresponse bias. To this end, 
reading test scores were regressed against total school enrollment obtained from the PIRLS questionnaire. 
There was a statistically significant linear relationship, with the school enrollment parameter estimate 
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having a p-value of 0.0039. A quadratic relationship was also tested, but the higher order term was not 
significant. The value of the school enrollment parameter estimate in the linear model was -0.043, 
indicating a negative relationship between reading test scores and school size. Combining the facts that 
responding schools tended to be smaller in size than nonresponding schools, and that smaller schools 
seemed to do better in the reading literacy tests, it is possible that the PIRLS results overestimate 
students’ reading abilities. 


4. CONCLUSIONS 


Westat’s investigation into nonresponse bias at the school level for PIRLS has shown that 
there is no statistically significant relationship between response status and the majority of school 
characteristics that were available for analysis. 

However, for the original sample of 200 schools, whether the school was public, private — 
Catholic, private — other religious, or private — non-sectarian, was a significant predictor of response 
status. Catholic schools were the most likely to respond, and private — other religious schools the least 
likely. Once replacements were used, this association was no longer apparent for the final sample of 200 
schools. 


The use of replacement schools did however seem to introduce a nonresponse bias that was 
not present in the original sample of schools. For the final sample, the number of students enrolled in 
grade 4 at the school was negatively related to response propensity. That is, schools with a higher number 
of students in grade 4 were less likely to respond. This effect may have been introduced if it was easier to 
get replacements to respond for smaller schools than it was for larger schools. 

It is difficult to assess the amount of any bias that may have been introduced into the survey 
results as a result of the association just described. However, investigations into the association between 
reading test scores and school size indicated that smaller schools tended to do statistically significantly 
better than larger schools, leaving the possibility that school nonresponse has resulted in an upward bias 
in results. 
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One way of approximately quantifying this is as follows. After replacements, the 
nonresponding schools make up 8 percent of the population (table 12). On average they have an 
enrollment that is 220 students higher than responding schools (table 16). The regression model indicates 
that each extra student is associated with a decrease of 0.043 in mean achievement score. Together these 
imply that the score for students from nonresponding schools might be about 9.5 points lower than for 
students from responding schools, so that the school nonresponse bias might be in the order of 0.8 scale 
score points. This is before any mitigating effects of nonresponse bias adjustments. Thus even though 
there is a statistically significant relationship between school size and response status in the final sample, 
it seems very likely to have had a negligible impact on overall study results. 
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94-03 1991 Schools and Staffing Survey (SASS) Reinterview Response Variance Report 

94—04 The Accuracy of Teachers’ Self-reports on their Postsecondary Education: Teacher 
Transcript Study, Schools and Staffing Survey 

94- 06 Six Papers on Teachers from the 1990-91 Schools and Staffing Survey and Other Related 

Surveys 

95- 01 Schools and Staffing Survey: 1994 Papers Presented at the 1994 Meeting of the American 

Statistical Association 

95-02 QED Estimates of the 1990-91 Schools and Staffing Survey: Deriving and Comparing 
QED School Estimates with CCD Estimates 

95-03 Schools and Staffing Survey: 1990-91 SASS Cross-Questionnaire Analysis 

95-08 CCD Adjustment to the 1990-91 SASS: A Comparison of Estimates 

95-09 The Results of the 1993 Teacher List Validation Study (TLVS) 

95-10 The Results of the 1991-92 Teacher Follow-up Survey (TFS) Reinterview and Extensive 

Reconciliation 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

95-12 Rural Education Data User’s Guide 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

95-15 Classroom Instructional Processes: A Review of Existing Measurement Approaches and 
Their Applicability for the Teacher Follow-up Survey 
95-16 Intersurvey Consistency in NCES Private School Surveys 

95- 18 An Agenda for Research on Teachers and Schools: Revisiting NCES’ Schools and 

Staffing Survey 

96- 01 Methodological Issues in the Study of Teachers’ Careers: Critical Features of a Truly 

Longitudinal Study 

96-02 Schools and Staffing Survey (SASS): 1995 Selected papers presented at the 1995 Meeting 
of the American Statistical Association 

96-05 Cognitive Research on the Teacher Listing Form for the Schools and Staffing Survey 

96-06 The Schools and Staffing Survey (SASS) for 1998-99: Design Recommendations to 
Inform Broad Education Policy 

96-07 Should SASS Measure Instructional Processes and Teacher Effectiveness? 

96-09 Making Data Relevant for Policy Discussions: Redesigning the School Administrator 
Questionnaire for the 1998-99 SASS 

96-10 1998-99 Schools and Staffing Survey: Issues Related to Survey Depth 


Laurence Ogle 

Marilyn Binkley 
Laurence Ogle 

Steven Kaufman 
Marilyn Seastrom 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 

Sharon Bobbitt & 
John Ralph 
Samuel Peng 
Samuel Peng 

Sharon Bobbitt 

Steven Kaufman 
Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 
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No. 

96-11 

96-12 

96-15 

96-23 

96-24 

96-25 

96- 28 

97- 01 

97-07 

97-09 

97-10 

97-11 

97-12 

97-14 

97-18 

97-22 

97-23 

97-41 

97- 42 
97^14 

98- 01 
98-02 
98-04 
98-05 

98-08 

98-12 

98-13 

98-14 

98-15 

98-16 

1999-02 

1999-04 

1999-07 

1999-08 

1999-10 


Title 

Towards an Organizational Database on America’s Schools: A Proposal for the Future of 
SASS, with comments on School Reform, Governance, and Finance 
Predictors of Retention, Transfer, and Attrition of Special and General Education 
Teachers: Data from the 1989 Teacher Followup Survey 
Nested Structures: District-Level Data in the Schools and Staffing Survey 
Linking Student Data to SASS: Why, When, Flow 
National Assessments of Teacher Quality 

Measures of Inservice Professional Development: Suggested Items for the 1998-1999 
Schools and Staffing Survey 

Student Learning, Teaching Quality, and Professional Development: Theoretical 
Linkages, Current Measurement, and Recommendations for Future Data Collection 
Selected Papers on Education Surveys: Papers Presented at the 1996 Meeting of the 
American Statistical Association 

The Dctenninants of Per-Pupil Expenditures in Private Elementary and Secondary 
Schools: An Exploratory Analysis 
Status of Data on Crime and Violence in Schools: Final Report 

Report of Cognitive Research on the Public and Private School Teacher Questionnaires 
for the Schools and Staffing Survey 1993-94 School Year 
International Comparisons of Inservice Professional Development 
Measuring School Reform: Recommendations for Future SASS Data Collection 
Optimal Choice of Periodicities for the Schools and Staffing Survey: Modeling and 
Analysis 

Improving the Mail Return Rates of SASS Surveys: A Review of the Literature 
Collection of Private School Finance Data: Development of a Questionnaire 
Further Cognitive Research on the Schools and Staffing Survey (SASS) Teacher Listing 
Form 

Selected Papers on the Schools and Staffing Survey: Papers Presented at the 1997 Meeting 
of the American Statistical Association 

Improving the Measurement of Staffing Resources at the School Level: The Development 
of Recommendations for NCES for the Schools and Staffing Survey (SASS) 
Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 
State Assessments and State NAEP, Feasibility Study 
Collection of Public School Expenditure Data: Development of a Questionnaire 
Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report 
Geographic Variations in Public Schools’ Costs 

SASS Documentation: 1993-94 SASS Student Sampling Problems; Solutions for 
Determining the Numerators for the SASS Private School (3B) Second-Stage Factors 
The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 
A Bootstrap Variance Estimator for Systematic PPS Sampling 
Response Variance in the 1994-95 Teacher Follow-up Survey 
Variance Estimation of Imputed Survey Data 

Development of a Prototype System for Accessing Linked NCES Data 
A Feasibility Study of Longitudinal Design for Schools and Staffing Survey 
Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results 
Measuring Teacher Qualifications 

Collection of Resource and Expenditure Data on the Schools and Staffing Survey 
Measuring Classroom Instructional Processes: Using Survey and Case Study Fieldtest 
Results to Improve Item Construction 
What Users Say About Schools and Staffing Survey Publications 


NCES contact 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 

Mary Rollefson 

Dan Kasprzyk 

Stephen Broughman 

Lee Hoffman 
Dan Kasprzyk 

Dan Kasprzyk 
Mary Rollefson 
Steven Kaufman 

Steven Kaufman 
Stephen Broughman 
Dan Kasprzyk 

Steve Kaufman 

Mary Rollefson 

Michael Ross 

Stephen Broughman 
Steven Kaufman 
William J. Fowler, Jr. 
Steven Kaufman 

Dan Kasprzyk 
Steven Kaufman 
Steven Kaufman 
Steven Kaufman 
Steven Kaufman 
Stephen Broughman 
Dan Kasprzyk 
Dan Kasprzyk 
Stephen Broughman 
Dan Kasprzyk 

Dan Kasprzyk 
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No. 

1999-12 

1999-13 

1999-14 

1999- 17 

2000- 04 

2000-10 

2000-13 

2000-18 

2002-04 


Title 

1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume III: Public-Use 
Codebook 

1993- 94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of 
Indian Affairs (BIA) Restricted-Use Codebook 

1994- 95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook 
Secondary Use of the Schools and Staffing Survey Data 

Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 

A Research Agenda for the 1999-2000 Schools and Staffing Survey 
Non-professional Staff in the Schools and Staffing Survey (SASS) and Common Core of 
Data (CCD) 

Feasibility Report: School-Level Finance Pretest, Public School District Questionnaire 
Improving Consistency of Response Categories Across NCES Surveys 


NCES contact 

Kerry Gruber 

Kerry Gruber 

Kerry Gruber 
Susan Wiley 
Dan Kasprzyk 

Dan Kasprzyk 
Kerry Gruber 

Stephen Broughman 
Marilyn Seastrom 


Third International Mathematics and Science Study (TIMSS) 

2001-01 Cross-National Variation in Educational Preparation for Adulthood: From Early 
Adolescence to Young Adulthood 

2001-05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 

2001- 07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 

International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 

2002- 01 Legal and Ethical Issues in the Use of Video in Education Research 


Elvira Hausken 

Patrick Gonzales 
Arnold Goldstein 

Patrick Gonzales 
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Listing of NCES Working Papers by Subject 


No. Title 


NCES contact 


Achievement (student) - mathematics 

2001-05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 

Adult education 

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult 
Education Component 

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96- 22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 
98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education 
Survey 

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks 
and Empirical Studies 

1999- 11 Data Sources on Lifelong Learning Available from the National Center for Education 

Statistics 

2000- 16a Lifelong Learning NCES Task Force: Final Report Volume I 

2000- 1 6b Lifelong Learning NCES Task Force: Final Report Volume II 

Adult literacy — see Literacy of adults 
American Indian - education 

1999-13 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of 

Indian Affairs (BIA) Restricted-Use Codebook 

Assessment/achievement 
95-12 Rural Education Data User’s Guide 

95-13 Assessing Students with Disabilities and Limited English Proficiency 

97- 29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? 

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable 

Assessment Results 

97-3 1 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational 
Progress 

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background 
Questions) 

97- 37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items 

97^14 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 
State Assessments and State NAEP, Feasibility Study 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

2001- 07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 

International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 

2001-1 1 Impact of Selected Background Variables on Students’ NAEP Math Performance 
2001-13 The Effects of Accommodations on the Assessment of LEP Students in NAEP 


Patrick Gonzales 

Steven Kaufman 

Kathryn Chandler 

Kathryn Chandler 

Peter Stowe 

Peter Stowe 

Lisa Hudson 

Lisa Hudson 
Lisa Hudson 

Kerry Gruber 

Samuel Peng 
James Houser 
Larry Ogle 
Larry Ogle 

Larry Ogle 

Larry Ogle 

Larry Ogle 
Michael Ross 

Jeffrey Owings 

Arnold Goldstein 

Arnold Goldstein 
Arnold Goldstein 
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No. Title 

2001- 19 The Measurement of Home Background Indicators: Cognitive Laboratory Investigations 

of the Responses of Fourth and Eighth Graders to Questionnaire Items and Parental 
Assessment of the Invasiveness of These Items 

2002- 05 Early Childhood Longitudinal Study-Kindergarten Class of 1998-99 (ECLS-K), 

Psychometric Report for Kindergarten Through First Grade 

2002- 06 The Measurement of Instructional Background Indicators: Cognitive Laboratory 

Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
Questionnaire Items 

2003- 19 NAEP Quality Assurance Checks of the 2002 Reading Assessment Results of Delaware 

Beginning students in postsecondary education 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

2001-04 Beginning Postsecondary Students Longitudinal Study: 1996-2001 (BPS: 1996/2001) 
Field Test Methodology Report 

Civic participation 

97-25 1996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 

Climate of schools 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

Cost of education indices 

94-05 Cost-of-Education Differentials Across the States 


Course-taking 

95-12 Rural Education Data User’s Guide 

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 
Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 
1999-05 Procedures Guide for Transcript Studies 
1999-06 1998 Revision of the Secondary School Taxonomy 

2003-01 Mathematics, Foreign Language, and Science Coursetaking and the NELS:88 Transcript 
Data 

2003-02 English Coursetaking and the NELS:88 Transcript Data 

Crime 

97- 09 Status of Data on Crime and Violence in Schools: Final Report 

Curriculum 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 


Customer service 


NCES contact 
Arnold Goldstein 


Elvira Hausken 
Arnold Goldstein 

Janis Brown 

Aurora D’Amico 
Paula Knepper 

Kathryn Chandler 


Samuel Peng 


William J. Fowler, Jr. 


Samuel Peng 
Jeffrey Owings 


Dawn Nelson 
Dawn Nelson 
Jeffrey Owings 

Jeffrey Owings 


Lee Hoffman 


Sharon Bobbitt & 
John Ralph 
Jeffrey Owings 
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No. Title NCES contact 

1999- 10 What Users Say About Schools and Staffing Survey Publications Dan Kasprzyk 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps Valena Plisko 

2000- 04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and Dan Kasprzyk 

1999 AAPOR Meetings 

Data quality 

97-13 Improving Data Quality in NCES: Database-to-Report Process 

2001- 1 1 Impact of Selected Background Variables on Students’ NAEP Math Performance 

2001-13 The Effects of Accommodations on the Assessment of LEP Students in NAEP 

2001- 19 The Measurement of Home Background Indicators: Cognitive Laboratory Investigations 

of the Responses of Fourth and Eighth Graders to Questionnaire Items and Parental 
Assessment of the Invasiveness of These Items 

2002- 06 The Measurement of Instructional Background Indicators: Cognitive Laboratory 

Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
Questionnaire Items 

2003- 19 NAEP Quality Assurance Checks of the 2002 Reading Assessment Results of Delaware 

Data warehouse 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and Dan Kasprzyk 
1999 AAPOR Meetings 

Design effects 

2000- 03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing Ralph Lee 

Variances from NCES Data Sets 

Dropout rates, high school 

95- 07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and Jeffrey Owings 

NELS:88 Sophomore Cohort Dropouts 

Early childhood education 

96- 20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early Kathryn Chandler 

Childhood Education, and Adult Education 

96- 22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early Kathryn Chandler 

Childhood Program Participation, and Adult Education 

97- 24 Formulating a Design for the ECLS: A Review of Longitudinal Studies Jerry West 

97- 36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood Jerry West 

Programs: A Review and Recommendations for Future Research 
1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West 

2001- 02 Measuring Father Involvement in Young Children's Lives: Recommendations for a Jerry West 

Fatherhood Module for the ECLS-B 

2001-03 Measures of Socio-Emotional Development in Middle School Elvira Hausken 

2001- 06 Papers from the Early Childhood Longitudinal Studies Program: Presented at the 2001 Jerry West 

AERA and SRCD Meetings 

2002- 05 Early Childhood Longitudinal Study-Kindergarten Class of 1998-99 (ECLS-K), Elvira Hausken 

Psychometric Report for Kindergarten Through First Grade 

Educational attainment 

98- 1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field Aurora D’Amico 

Test Report 


Susan Ahmed 
Arnold Goldstein 
Arnold Goldstein 
Arnold Goldstein 


Arnold Goldstein 


Janis Brown 
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No. Title NCES contact 

2001- 15 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test Andrew G. Malizio 

Methodology Report 

Educational research 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps Valena Plisko 

2002- 01 Legal and Ethical Issues in the Use of Video in Education Research Patrick Gonzales 

Eighth-graders 

2001- 05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics Patrick Gonzales 

Employment 

96- 03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and Issues 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 

Test Report 

2000-16a Lifelong Learning NCES Task Force: Final Report Volume I 

2000- 1 6b Lifelong Learning NCES Task Force: Final Report Volume II 

2001- 01 Cross-National Variation in Educational Preparation for Adulthood: From Early 

Adolescence to Young Adulthood 

Employment - after college 

2001-15 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test Andrew G. Malizio 

Methodology Report 

Engineering 

2000- 1 1 Financial Aid Profile of Graduate Students in Science and Engineering Aurora D’Amico 

Enrollment - after college 

2001- 15 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test Andrew G. Malizio 

Methodology Report 

Faculty - higher education 

97- 26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

2000- 01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

2002- 08 A Profile of Part-time Faculty: Fall 1998 

Fathers - role in education 

2001- 02 Measuring Father Involvement in Young Children's Lives: Recommendations for a Jerry West 

Fatherhood Module for the ECLS-B 

Finance - elementary and secondary schools 

94-05 Cost-of-Education Differentials Across the States 

96- 19 Assessment and Analysis of School-Level Expenditures 

98- 01 Collection of Public School Expenditure Data: Development of a Questionnaire 

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

1999- 16 Measuring Resources in Education: From Accounting to the Resource Cost Model 

Approach 

2000- 18 Feasibility Report: School-Level Finance Pretest, Public School District Questionnaire 

Finance - postsecondary 

97- 27 Pilot Test of IPEDS Finance Survey Peter Stowe 


William J. Fowler, Jr. 
William J. Fowler, Jr. 
Stephen Broughman 
Stephen Broughman 
William J. Fowler, Jr. 

Stephen Broughman 


Linda Zimbler 
Linda Zimbler 
Linda Zimbler 


Jeffrey Owings 
Aurora D’Amico 

Lisa Hudson 
Lisa Hudson 
Elvira Hausken 
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No. Title 

2000-14 IPEDS Finance Data Comparisons Under the 1997 Financial Accounting Standards for 
Private, Not-for-Profit Institutes: A Concept Paper 


Finance - private schools 

95- 17 Estimates of Expenditures for Private K-12 Schools 

96- 16 Strategies for Collecting Finance Data from Private Schools 

97- 07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 

Schools: An Exploratory Analysis 

97-22 Collection of Private School Finance Data: Development of a Questionnaire 

1999- 07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

2000- 15 Feasibility Report: School-Level Finance Pretest, Private School Questionnaire 


Geography 

98-04 Geographic Variations in Public Schools’ Costs 


Graduate students 

2000-1 1 Financial Aid Profile of Graduate Students in Science and Engineering 


Graduates of postsecondary education 

2001-15 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test 
Methodology Report 


Imputation 

2000- 04 

2001 - 10 
2001-16 
2001-17 
2001-18 
2003-20 


Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meeting 

Comparison of Proc Impute and Schafer’s Multiple Imputation Software 
Imputation of Test Scores in the National Education Longitudinal Study of 1988 
A Study of Imputation Algorithms 
A Study of Variance Estimation Methods 

Imputation Methodology for the National Postsecondary Student Aid Study: 2004 


Inflation 

97 — 43 Measuring Inflation in Public School Costs 

Institution data 

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 


Instructional resources and practices 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test 
Results to Improve Item Construction 


International comparisons 

97-1 1 International Comparisons of Inservice Professional Development 

97-16 International Education Expenditure Comparability Study: Final Report, Volume I 


NCES contact 
Peter Stowe 


Stephen Broughman 
Stephen Broughman 
Stephen Broughman 

Stephen Broughman 
Stephen Broughman 
Stephen Broughman 

William J. Fowler, Jr. 

Aurora D’Amico 


Andrew G. Malizio 


Dan Kasprzyk 

Sam Peng 
Ralph Lee 
Ralph Lee 
Ralph Lee 
James Griffith 


William J. Fowler, Jr. 
Linda Zimbler 


Sharon Bobbitt & 
John Ralph 
Dan Kasprzyk 


Dan Kasprzyk 
Shelley Bums 


34 



No. Title 

97-17 International Education Expenditure Comparability Study: Final Report, Volume II, 
Quantitative Analysis of Expenditure Comparability 
2001-01 Cross-National Variation in Educational Preparation for Adulthood: From Early 
Adolescence to Young Adulthood 

2001-07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 
International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 


International comparisons - math and science achievement 

2001-05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 

Libraries 

94- 07 Data Comparability and Public Policy: New Interest in Public Library Data Papers 

Presented at Meetings of the American Statistical Association 
97-25 1996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 

Limited English Proficiency 

95- 13 Assessing Students with Disabilities and Limited English Proficiency 

2001-1 1 Impact of Selected Background Variables on Students’ NAEP Math Performance 
2001-13 The Effects of Accommodations on the Assessment of LEP Students in NAEP 


Literacy of adults 

98-17 Developing the National Assessment of Adult Literacy: Recommendations from 
Stakeholders 

1999-09a 1992 National Adult Literacy Survey: An Overview 

1999-09b 1992 National Adult Literacy Survey: Sample Design 

1999-09c 1992 National Adult Literacy Survey: Weighting and Population Estimates 

1999-09d 1992 National Adult Literacy Survey: Development of the Survey Instruments 

1999-09e 1992 National Adult Literacy Survey: Scaling and Proficiency Estimates 

1999-09f 1992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy 

Levels 

1999-09g 1992 National Adult Literacy Survey: Literacy Levels and the Response Probability 

Convention 

1999- 11 Data Sources on Lifelong Learning Available from the National Center for Education 

Statistics 

2000- 05 Secondary Statistical Modeling With the National Assessment of Adult Literacy: 

Implications for the Design of the Background Questionnaire 
2000-06 Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-Door 
Surveys in the Assessment of Adult Literacy 

2000-07 "How Much Literacy is Enough?” Issues in Defining and Reporting Performance 
Standards for the National Assessment of Adult Literacy 
2000-08 Evaluation of the 1992 NALS Background Survey Questionnaire: An Analysis of Uses 
with Recommendations for Revisions 

2000- 09 Demographic Changes and Literacy Development in a Decade 

2001- 08 Assessing the Lexile Framework: Results of a Panel Meeting 


Literacy of adults - international 

97-33 Adult Literacy: An International Perspective 


NCES contact 
Shelley Burns 

Elvira Hausken 

Arnold Goldstein 


Patrick Gonzales 


Carrol Kindel 
Kathryn Chandler 


James Houser 
Arnold Goldstein 
Arnold Goldstein 


Sheida White 

Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 

Alex Sedlacek 

Lisa Hudson 

Sheida White 

Sheida White 

Sheida White 

Sheida White 

Sheida White 
Sheida White 


Marilyn Binkley 
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No. 

Title 

NCES contact 

Mathematics 


98-09 

High School Curriculum Structure: Effects on Coursetaking and Achievement in 
Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Jeffrey Owings 

1999-08 

Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test 
Results to Improve Item Construction 

Dan Kasprzyk 

2001-05 

Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 

Patrick Gonzales 

2001-07 

A Comparison of the National Assessment of Educational Progress (NAEP), the Third 
International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 

Arnold Goldstein 

2001-11 

2002-06 

Impact of Selected Background Variables on Students’ NAEP Math Performance 

The Measurement of Instructional Background Indicators: Cognitive Laboratory 

Arnold Goldstein 


Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
Questionnaire Items 


Parental involvement in education 

96- 03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and Jeffrey Owings 

Issues 

97- 25 1996 National Household Education Survey (NHES:96) Questionnaires: Kathryn Chandler 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 
1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West 

2001-06 Papers from the Early Childhood Longitudinal Studies Program: Presented at the 2001 Jerry West 

AERA and SRCD Meetings 

2001-19 The Measurement of Home Background Indicators: Cognitive Laboratory Investigations Arnold Goldstein 
of the Responses of Fourth and Eighth Graders to Questionnaire Items and Parental 
Assessment of the Invasiveness of These Items 

Participation rates 

98- 10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks Peter Stowe 

and Empirical Studies 

Postsecondary education 

1999- 11 Data Sources on Lifelong Learning Available from the National Center for Education Lisa Hudson 

Statistics 

2000- 16a Lifelong Learning NCES Task Force: Final Report Volume I Lisa Hudson 

2000-1 6b Lifelong Learning NCES Task Force: Final Report Volume II Lisa Hudson 

2003-20 Imputation Methodology for the National Postsecondary Student Aid Study: 2004 James Griffith 

Postsecondary education - persistence and attainment 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field Aurora D’Amico 
Test Report 

1999- 15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico 

Postsecondary education - staff 

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler 

2000- 01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler 


36 



NCES contact 
Linda Zimbler 


No. Title 

2002-08 A Profile of Part-time Faculty: Fall 1998 

Principals 

2000-10 A Research Agenda for the 1999-2000 Schools and Staffing Survey Dan Kasprzyk 

Private schools 

96- 16 Strategies for Collecting Finance Data from Private Schools 

97- 07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 

Schools: An Exploratory Analysis 

97-22 Collection of Private School Finance Data: Development of a Questiomiaire 
2000-13 Non-professional Staff in the Schools and Staffing Survey (SASS) and Common Core of 
Data (CCD) 

2000-15 Feasibility Report: School-Level Finance Pretest, Private School Questionnaire 

Projections of education statistics 

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico 

Public school finance 

1999- 16 Measuring Resources in Education: From Accounting to the Resource Cost Model Approach William J. Fowler, Jr. 

2000- 18 Feasibility Report: School-Level Finance Pretest, Public School District Questionnaire Stephen Broughman 

Public schools 

97 — 43 Measuring Inflation in Public School Costs 

98- 01 Collection of Public School Expenditure Data: Development of a Questionnaire 

98-04 Geographic Variations in Public Schools’ Costs 

1999- 02 Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results 

2000- 12 Coverage Evaluation of the 1994-95 Public Elementary/Secondary School Universe 

Survey 

2000-13 Non-professional Staff in the Schools and Staffing Survey (SASS) and Common Core of 
Data (CCD) 

2002-02 Locale Codes 1987 - 2000 

Public schools - secondary 

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in Jeffrey Owings 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Reform, educational 

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and Jeffrey Owings 

Issues 

Response rates 

98-02 Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report Steven Kaufman 

School districts 

2000-10 A Research Agenda for the 1999-2000 Schools and Staffing Survey Dan Kasprzyk 

School districts, public 

98-07 Decennial Census School District Project Planning Report Tai Phan 


William J. Fowler, Jr. 
Stephen Broughman 
William J. Fowler, Jr. 
Dan Kasprzyk 
Beth Young 

Kerry Gruber 

Frank Johnson 


Stephen Broughman 
Stephen Broughman 

Stephen Broughman 
Kerry Gruber 

Stephen Broughman 
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No. Title NCES contact 

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection, Beth Young 
Processing, and Editing Cycle 

School districts, public - demographics of 

96-04 Census Mapping Project/School District Data Book Tai Phan 


Schools 

97 — 42 Improving the Measurement of Staffing Resources at the School Level: The Development 

of Recommendations for NCES for the Schools and Staffing Survey (SASS) 

98- 08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

1999- 03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection, 

Processing, and Editing Cycle 

2000- 10 A Research Agenda for the 1999-2000 Schools and Staffing Survey 

2002-02 Locale Codes 1 987 - 2000 

Schools - safety and discipline 

97- 09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman 

Science 

2000- 1 1 Financial Aid Profile of Graduate Students in Science and Engineering 

2001- 07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 

International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 

Software evaluation 

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing Ralph Lee 
Variances from NCES Data Sets 

Staff 

97^12 Improving the Measurement of Staffing Resources at the School Level: The Development Mary Rollefson 

of Recommendations for NCES for the Schools and Staffing Survey (SASS) 

98- 08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk 

Staff - higher education institutions 

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

2002- 08 A Profile of Part-time Faculty: Fall 1998 

Staff - nonprofessional 

2000-13 Non-professional Staff in the Schools and Staffing Survey (SASS) and Common Core of Kerry Gruber 
Data (CCD) 

State 

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection, Beth Young 
Processing, and Editing Cycle 

2003- 19 NAEP Quality Assurance Checks of the 2002 Reading Assessment Results of Delaware Janis Brown 


Linda Zimbler 
Linda Zimbler 


Aurora D’Amico 
Arnold Goldstein 


Mary Rollefson 

Dan Kasprzyk 
Beth Young 

Dan Kasprzyk 
Frank Johnson 
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No. 


Title 


NCES contact 


Statistical methodology 

97-21 Statistics for Policymakers or Everything You Wanted to Know About Statistics But 
Thought You Could Never Understand 

2003-20 Imputation Methodology for the National Postsecondary Student Aid Study: 2004 

Statistical standards and methodology 

2001- 05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 

2002- 04 Improving Consistency of Response Categories Across NCES Surveys 

Students with disabilities 

95- 13 Assessing Students with Disabilities and Limited English Proficiency 

2001-13 The Effects of Accommodations on the Assessment of LEP Students in NAEP 

Survey methodology 

96- 17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report 

97- 15 Customer Service Survey: Common Core of Data Coordinators 

97- 35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1996 

National Household Education Survey 

98- 06 National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second 

Follow-Up: Final Methodology Report 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

98-16 A Feasibility Study of Longitudinal Design for Schools and Staffing Survey 
1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

1999- 17 Secondary Use of the Schools and Staffing Survey Data 

2000- 01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 

2000-12 Coverage Evaluation of the 1994-95 Public Elementary/Secondary School Universe 
Survey 

2000- 17 National Postsecondary Student Aid Study:2000 Field Test Methodology Report 

2001- 04 Beginning Postsecondary Students Longitudinal Study: 1996-2001 (BPS: 1996/200 1 ) 

Field Test Methodology Report 

2001-07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 
International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 

2001-1 1 Impact of Selected Background Variables on Students’ NAEP Math Performance 
2001-13 The Effects of Accommodations on the Assessment of LEP Students in NAEP 

2001- 19 The Measurement of Home Background Indicators: Cognitive Laboratory Investigations 

of the Responses of Fourth and Eighth Graders to Questionnaire Items and Parental 
Assessment of the Invasiveness of These Items 

2002- 01 Legal and Ethical Issues in the Use of Video in Education Research 

2002-02 Locale Codes 1987 - 2000 

2002-03 National Postsecondary Student Aid Study, 1999-2000 (NPSAS:2000), CATI 
Nonresponse Bias Analysis Report. 


Susan Ahmed 
James Griffith 


Patrick Gonzales 
Marilyn Seastrom 

James Houser 
Arnold Goldstein 


Andrew G. Malizio 
Lee Hoffman 
Kathryn Chandler 

Ralph Lee 

Aurora D’Amico 

Stephen Broughman 
Stephen Broughman 
Susan Wiley 
Linda Zimbler 
Valena Plisko 
Dan Kasprzyk 

Beth Young 

Andrew G. Malizio 
Paula Knepper 

Arnold Goldstein 


Arnold Goldstein 
Arnold Goldstein 
Arnold Goldstein 


Patrick Gonzales 
Frank Johnson 
Andrew Malizio 
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No. Title 

2002- 06 The Measurement of Instructional Background Indicators: Cognitive Laboratory 

Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
Questionnaire Items 

2003- 03 Education Longitudinal Study: 2002 (ELS: 2002) Field Test Report 

2003-21 U.S. 2001 PIRLS Nonresponse Bias Analysis 

Teachers 

98-13 Response Variance in the 1994—95 Teacher Follow-up Survey 

1999- 14 1994—95 Teacher Followup Survey: Data File User’s Manual, Restricted -Use Codebook 

2000- 10 A Research Agenda for the 1999-2000 Schools and Staffing Survey 

Teachers - instructional practices of 

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

2002- 06 The Measurement of Instructional Background Indicators: Cognitive Laboratory 

Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
Questionnaire Items 

Teachers - opinions regarding safety 

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

Teachers - performance evaluations 

1999-04 Measuring Teacher Qualifications 

Teachers - qualifications of 

1999- 04 Measuring Teacher Qualifications 

Teachers - salaries of 

94-05 Cost-of-Education Differentials Across the States 

Training 

2000- 16a Lifelong Learning NCES Task Force: Final Report Volume I 

2000-1 6b Lifelong Learning NCES Task Force: Final Report Volume II 

Variance estimation 

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 
Variances from NCES Data Sets 

2000- 04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 

1999 AAPOR Meetings 

200 1- 18 A Study of Variance Estimation Methods 

2003- 18 Report for Computation of Balanced Repeated Replicate (BRR) Weights for the Third 

(NELS88:1994) and Fourth (NELS88:2000) Follow-up Surveys 
2003-20 Imputation Methodology for the National Postsecondary Student Aid Study: 2004 


Violence 

97-09 Status of Data on Crime and Violence in Schools: Final Report 


NCES contact 
Arnold Goldstein 


Jeffrey Owings 
Laurence Ogle 

Steven Kaufman 
Kerry Gruber 
Dan Kasprzyk 

Dan Kasprzyk 
Arnold Goldstein 


Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

William J. Fowler, Jr. 

Lisa Hudson 
Lisa Hudson 

Ralph Lee 

Dan Kasprzyk 

Ralph Lee 
Dennis Carroll 

James Griffith 


Lee Hoffman 


40 



No. 


Title 


NCES contact 


Vocational education 

95-12 Rural Education Data User’s Guide 
1999-05 Procedures Guide for Transcript Studies 
1999-06 1998 Revision of the Secondary School Taxonomy 


Samuel Peng 
Dawn Nelson 
Dawn Nelson 
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